fix(runtime): four more process-globals that contradict the multi-state contract (#739 item 4) - #764
Conversation
…#739 item 3) The cooperative scheduler lives on EigsThread — tasks are single-threaded by construction — but the suspend request that drives it was a plain global, set at four sites and polled by every vm_run on every thread at CASE(CALL). So a task_yield on the main thread drove an unrelated OS worker's next call into vm_suspend_halt. Having no scheduler of its own, the victim's task_save_slice(task_current_running()) was handed NULL and saved nothing, and vm_run returned NULL mid-evaluation with its frames deliberately undrained (the suspend path preserves them on purpose). A spawned worker therefore produced `null` instead of its result, silently. Reproduced with a control — two workers each computing 400000 while main drives a task_yield loop: workers + main yielding w1=null w2=null (3 of 3 runs) identical program, no yields w1=400000 w2=400000 (3 of 3 runs) The flag is now per-thread. It sits on EigsThread beside the task_sched it drives rather than inside TaskScheduler as the issue sketched: the poll is on the hot CASE(CALL) path, and inside the scheduler it would need `g_task_sched && ((TaskScheduler*)g_task_sched)->suspend_request` — two dependent loads plus a branch, paid by every thread that has no scheduler at all, which is nearly all of them. On EigsThread it stays a single load off the already-hot eigs_current, no NULL check. tests/test_tasks.eigs gains the case; the suite had none where cooperative tasks and OS threads ran at the same time. Validated red: "expected 400000, got null" on both workers without the fix. Gates: release 3274/3274; asan+ubsan 3278/3278, leak tally 0; TSan 11/11 including its seeded-race self-validation (the gate is live); tasks 79/79; freestanding stage 1+2 OK; jit-smoke; embed stack soak OK. Refs #739 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…te contract (#739 item 4) Each is a resource the runtime scopes per state or per thread everywhere else, held in one process-wide slot. - s_osr_threshold (vm.c) was a FUNCTION STATIC filled by whichever state crossed a back edge first, then frozen for the life of the process — defeating the per-state JIT tuning eigenscript.h explicitly advertises ("so two co-located embedded states can tune independently"). Entry and iter thresholds were already per-state; only OSR opted out. It now reads g_osr_threshold directly. eigs_jit_get_osr_threshold() existed only to fill that cache and is deleted; its comment claimed the value was read "once per thread", which was never true — once per PROCESS. - g_sandbox_active / _bytes_used / _byte_max / g_sandbox_loop_max are now per-thread. The save/restore in builtin_sandbox_run is correct for one thread's nesting, but the premise it documented — "sandbox_run is synchronous" — holds per-thread and breaks the moment two states run concurrently, which ext_http does per connection: one worker entering a sandbox capped every other worker's loops and charged their allocations against its budget. - g_stream_file is now per-thread and closed at eigs_thread_detach. One FILE* per process meant stream_open's unconditional close tore down another state's in-flight stream; an unclosed stream also lost its buffered tail at exit instead of being flushed (verified: 24 bytes now reach disk where they previously did not). The cleanup is carved out under !EIGENSCRIPT_FREESTANDING with the stream_* builtins themselves — the freestanding symbol gate caught fclose leaving the allowlist. - g_db_conn moved onto EigsState, closed by a new ext_db_state_destroy called from eigs_state_destroy — the shape ext_http_state_destroy already had. One worker's db_connect used to replace the connection another worker was querying through, and nothing ever closed it. NOT BUILT OR RUN: this box has no libpq (no headers, no library), so ext_db.c and state.c with EXT_DB=1 are syntax-checked against a local stub header only. Worth a careful look in review. g_model is deliberately left process-wide. Its allocations are reachable from a global at exit, so they are still-reachable rather than leaked — LSan does not report them. The genuine defect would be cross-state clobber, and per-state transformer weights multiply memory by the number of states; that is a design decision about model isolation, not a mechanical scope move. Same for g_model_age / g_training_samples. Gates: release 3274/3274; asan+ubsan 3278/3278, leak tally 0; TSan 11/11 including its seeded-race self-validation (17 warnings — the gate is live); freestanding stage 1+2 OK (after the carve-out); jit-smoke; sandbox + task + HTTP suites pass. Refs #739 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
This PR completes the remaining “item 4” fixes from #739 by eliminating several process-global resources that break the runtime’s multi-state / multi-thread contract, and by documenting + regression-testing the task-yield cross-thread suspend bug.
Changes:
- Remove the function-static OSR threshold cache so JIT OSR tuning remains per-state; delete the now-unneeded
eigs_jit_get_osr_threshold()accessor. - Move sandbox caps/budget and
stream_open’s target stream from process globals to per-thread (EigsThread) storage, with thread-detach cleanup for unclosed streams. - Move the DB connection from process-global to per-state (
EigsState) storage and close it at state teardown; update docs/spec/changelog and add a regression test for the task-yield bug.
Reviewed changes
Copilot reviewed 15 out of 15 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/test_tasks.eigs | Adds a regression test ensuring task_yield on one OS thread cannot suspend other OS threads. |
| src/vm.h | Updates comments to reflect task-suspend and sandbox controls now being per-thread bridge macros. |
| src/vm.c | Removes process-global sandbox/task-suspend variables; reads OSR threshold directly from per-state tuning. |
| src/state.c | Calls ext_db_state_destroy on state teardown; closes per-thread stream handle on thread detach (hosted builds). |
| src/jit.h | Removes eigs_jit_get_osr_threshold() declaration (no longer used). |
| src/jit.c | Removes eigs_jit_get_osr_threshold() definition (no longer needed). |
| src/ext_db.c | Adds ext_db_state_destroy to close the per-state DB connection. |
| src/ext_db_internal.h | Replaces the process-global DB connection with a per-state accessor macro. |
| src/eigenscript.h | Adds per-thread fields for sandbox/task/stream and exposes new bridge macros; adds per-state ext_db_conn. |
| src/builtins.c | Switches stream builtins from a process-global FILE* to the per-thread bridge macro. |
| docs/SPEC.md | Documents that cooperative tasks are scoped per OS thread. |
| docs/EMBEDDING.md | Documents per-state/per-thread teardown responsibilities for embedding. |
| docs/CONCURRENCY.md | Adds explicit documentation that cooperative tasks (and suspend requests) are per-thread. |
| docs/BUILTINS.md | Documents stream_open as “one stream per thread” and that unclosed streams are closed at thread end. |
| CHANGELOG.md | Records the fixed process-global violations and the task-yield cross-thread suspend bug. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| (`ext_http_state_destroy`, `ext_db_state_destroy`). Sandbox budgets, | ||
| the `stream_open` file handle, the JIT's OSR threshold, and the | ||
| cooperative task scheduler are per-**thread** and released at | ||
| `eigs_thread_detach`. Two co-located states therefore do not share a |
| #define g_sandbox_loop_max (eigs_current->sandbox_loop_max) | ||
| #define g_sandbox_active (eigs_current->sandbox_active) | ||
| #define g_sandbox_bytes_used (eigs_current->sandbox_bytes_used) | ||
| #define g_sandbox_byte_max (eigs_current->sandbox_byte_max) | ||
| #define g_stream_file (*(FILE **)&eigs_current->stream_file) |
| /* #739: per-STATE connection, reached through the attached thread — the same | ||
| * shape ext_http's per-state Server uses. Defined here rather than in | ||
| * eigenscript.h so libpq's types stay out of the core header. */ | ||
| #define g_db_conn (*(PGconn **)&eigs_current->state->ext_db_conn) | ||
|
|
|
Correction: this PR also carried #739 item 3, which its description did not mention. I branched Nobody reading this PR's description would have expected the task-scheduler changes in it. The work itself is the same work that was up for review in #763 — same commit, same gates, same red-then-green validation ( #763 is now redundant and I am closing it with a pointer here. The item-3 rationale, including why the flag went on |
Last of #739's items. Item 1 (trace tape) → #761, item 2 (
exitrequest) → #762, item 3 (task suspend) → #763.Each of these is a resource the runtime scopes per state or per thread everywhere else, held in one process-wide slot.
s_osr_threshold— a function static froze per-state JIT tuningIt was filled by whichever state crossed a back edge first, then frozen for the life of the process — defeating the tuning
eigenscript.hexplicitly advertises ("so two co-located embedded states can tune independently"). Entry and iter thresholds were already per-state; only OSR opted out. It now readsg_osr_thresholddirectly.eigs_jit_get_osr_threshold()existed only to fill that cache and is deleted. Its comment claimed the value was read "once per thread" — never true; it was once per process, which is precisely the bug.g_sandbox_*— one budget shared by every concurrent stateg_sandbox_active/_bytes_used/_byte_max/g_sandbox_loop_maxare now per-thread. The save/restore inbuiltin_sandbox_runis correct for one thread's nesting, but the premise it documented — "sandbox_run is synchronous" — holds per thread and breaks the moment two states run concurrently, whichext_httpdoes per connection. One worker entering a sandbox capped every other worker's loops and charged their allocations against its budget.g_stream_file— one FILE* per processNow per-thread, closed at
eigs_thread_detach.stream_openunconditionally closes whatever is already open, so two states streaming at once tore down each other's file mid-write. An unclosed stream also lost its buffered tail at exit; verified 24 bytes now reach disk where they previously did not.The freestanding symbol gate caught
fcloseleaving the allowlist, so the cleanup is carved out under!EIGENSCRIPT_FREESTANDINGalongside thestream_*builtins themselves. Worth noting the gate did exactly its job here.g_db_conn— per-state, and actually closedMoved onto
EigsState, closed by a newext_db_state_destroycalled fromeigs_state_destroy— the same shapeext_http_state_destroyalready had. One worker'sdb_connectused to replace the connection another worker was querying through, and nothing ever closed it.ext_db.candstate.cwithEXT_DB=1are syntax-checked against a local stub header only. That catches compile errors and proves nothing about behavior. This is the one hunk that deserves real review.Deliberately unchanged:
g_modelIts allocations are reachable from a global at exit, so they are still-reachable, not leaked — LSan does not report them and the OS reclaims them. The genuine defect would be cross-state clobber (one state's
eigen_model_loadreplacing another's weights), and the fix for that is per-state transformer weights, which multiplies memory by the number of states. That is a design decision about model isolation, not a mechanical scope move, so I left it rather than trade a non-leak for a memory regression. Same forg_model_age/g_training_samples.Validation
One transient to record rather than hide: a single
test_http_server.shrun showed 41/1 during this work. I did not capture which check, and five consecutive runs since are 42/0. Logging it as an observed transient, not as nothing.Refs #739 — with this, all four items are addressed. The intermittent
db extensionRSS-growth failure is not explained by any of them; I retracted my earlier hypothesis in the issue thread and proposed the diagnostic that separates the two variables that job conflates.🤖 Generated with Claude Code